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SUMMARY 
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Problem  and  Background 


The  Navy  Enlisted  Advancement  System  is  based  on  several  components 
which  are  differentially  weighted  in  arriving  at  a  final  ordered  list¬ 
ing  of  the  candidates.  The  components  used--technical  knowledge  exam, 
on- job  performance  marks,  time  in  grade  and  in  the  Navy,  and  awards — 
vary  greatly  in  their  statistical  properties,  and  thus  the  task  of  com¬ 
bining  them  in  accordance  with  the  policy-designated  weights  becomes 
complicated  and  difficult.  The  outcome  of  the  advancement  process-- 
that  is,  the  actual  selection  of  those  to  be  advanced--has  been  found 
to  be  more  a  function  of  the  statistical  peculiarities  of  the  compo¬ 
nents  used  and  of  the  methods  used  in  combining  them,  than  of  published 
policy. 


The  proportionate  influences  of  the  components  were  found  to  vary 
widely  as  a  function  of  advancement  opportunity,  cutting  scores  used, 
and  pay  grade. 


Short-term  recommendations  for  improving  the  present  system  were 
made  on  the  basis  of  the  present  analysis.  However,  basic  long-range 
improvement  in  the  system  must  await  the  findings  of  research  on  the 
differential  relevance  of  the  various  components  to  future  performance 
and  the  development  of  responsive  iterative  computer  programs,  for 
weighting  the  advancement  factors  in  accordance  with  policy. 


_ 


This  report  describes  a  comparison  of  several  approaches  to  com¬ 
bining  scores  and  weighting  components  used  in  the  advancement  compe¬ 
tition  to  Pay  Grades  H4-E7.  The  qualifications  of  candidates  who  were 
selected,  and  who  would  have  been  selected  had  other  cutting  scores  and 
weighting  methods  been  used,  were  compared.  The  investigation  repre¬ 
sents  an  attempt  to  understand  the  problems  inherent  in  the  system  and 
to  develop  procedures  to  increase  the  likelihood  that  the  advancement 
system  will  consistently  select  those  individuals  most  deserving  of 
advancement.  Because  of  its  influence  upon  morale,  the  policy  of  grant 
ing  "bonus  points"  to  candidates  who  pass  the  exsun  but  are  not  advanced 
(PNA)  was  in  particular  addressed. 
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COMPARISON  OF  ALTERNATIVE  CRITERIA  AND  WEIGHTING 
METHODS  FOR  THE  ENLISTED  ADVANCEMENT  SYSTEM 


A.  STATEMENT  OF  THE  PROBLEM 


1.  Background 

The  Enlisted  Advancement  System  is  a  vitally  important  personnel 
management  function  which  the  Navy  desires  to  improve  in  order  to  en¬ 
hance  morale  in  general  and  retention  in  particular.  Prior  to  imple¬ 
menting  changes  designed  to  effect  such  improvement,  research  must  be 
done  to: 


a.  Identify  some  of  the  alternative  courses  of  action  which  might 
be  considered, 

b.  Develop  analytical  methodologies  appropriate  for  investigating 
the  courses  of  action,  and 

c.  Determine  the  outcome  of  each  alternative. 

A  number  of  factors  are  considered  in  Navy  advancement  selections, 
and  these  are  differentially  weighted  in  an  advancement  composite,  the 
Final  Multiple  Score  (FMS) .  It  has  long  been  recognized  (e.g.,  the 
SECNAV  Task  Force  on  Retention,  1966)  that  the  appropriateness  and 
weighting  of  the  factors  comprising  this  advancement  multiple  have  an 
important  bearing  on  the  morale  of  enlisted  personnel. 

A  particularly  crucial  aspect  of  the  advancement  problem  as  it  re¬ 
lates  to  retention  is  the  disappointment  experienced  by  nonadvanced 
personnel  who  pass  the  advancement  exam  but  are  not  advanced  because 
of  quota  limitations  (i.e.,  Passed  but  Not  Advanced — PNA).  In  recogni¬ 

tion  of  this  situation,  which  is  prevalent  in  competition  to  the  higher 
pay  grades,  it  has  sometimes  been  recommended  that  "bonus  points"  be 
added  to  an  individual's  Final  Multiple  Score  for  each  time  the  candi¬ 
date  passes  an  examination  but  is  "PNA'd"  (e.g.,  Item  58  of  the  Career 
Motivation  Conference  of  1969). 

2.  Purpose 

This  report  presents  comparisons  among  alternative  analytical  tech¬ 
niques,  as  well  as  alternative  weighting  designs.  An  additional  interest 
is  also  to  identify  possible  actions  for  short-term  relief  of  specific 
problems,  pending  further  progress  on  long-term  developments.  Accord¬ 
ingly,  this  report  is  addressed  to  the  following  problems: 

a.  How  useful  are  the  various  available  methodologies  for  analyzing 
the  system  and  demonstrating  outcomes? 
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b.  What  is  the  incidence  of  PNA  and  exam-fail  in  different 
Ratings  and  pay  grades? 

c.  What  outcomes  derive  from  the  alternative  weighting  procedures? 

d.  What  short-term  possibilities  warrant  exploration  as  means  of 
improving  the  Enlisted  Advancement  System,  pending  the  development  of 
long-term  solutions? 


B.  PROCEDURES 


1.  Data 


The  factors  in  the  advancement  mu  tiple  include  a  written  exam 
score,  on-job  performance  evaluation  marks,  time  in  the  Navy,  time  in 
pay  grade  and  medals  and  awards.  Data  comprising  such  factors  were 
provided  by  the  Naval  Examining  Center,  Great  Lakes,  for  several  sam¬ 
ples  of  candidates: 

a.  From  the  group  of  candidates  who  competed  for  advancement  in 
February  1967  (Exam  Series  42),  a  period  of  relatively  high  active 
duty  force  levels,  the  following  representative  Ratings  were  selected 
for  analysis  (See  Appendix  B  for  complete  titles): 

Pay  Grade  Ratings 

4,  5,  6,  7  ADR,  AK,  AX,  CS,  DC,  ET,  FT,  HM,  PN,  RM,  ST,  TM 

b.  For  a  period  shortly  after  a  substantial  reduction  in  force 
level,  August  1969  (Exam  Series  51),  the  following  representative 
Ratings  were  selected: 


Pay  Grade 


4 

ABH. 

AX, 

5 

ABH, 

ADJ 

6 

ABH, 

ADJ 

7 

ABH, 

ADJ 

Ratings 

HM,  PN 

AX,  FT,  HM,  PN,  SD 

AT,  AX,  FT,  HM,  NN, 

AT,  AX,  EM,  FT,  HM, 


PN,  RM,  SD,  TM 

MM,  W,  PN,  RM,  SD,  TM 


In  order  to  construct  a  pass-fail  history,  data  for  all  advancement 
candidates  were  utilized  for  the  above  Ratings  for  all  exam  series  in 
calendar  years  '67,  '68  and  '69.  These  data  included  two  series  per 
year  to  Pay  Grades  5,  6  and  7;  and  four  series  to  Pay  Grade  4  in  1969. 
Ratings  selected  for  analysis  were  those  which  varied  widely  in  advance¬ 
ment  opportunity  and  PNA  incidence. 

c.  Since  the  Series  51  data  were  the  most  recent  available  at  the 
time  of  commencing  the  analysis,  in  order  to  make  the  study  relevant 
to  current  conditions,  Series  55  (August  1970)  decisions  were  simulated 
by  applying  Series  55  advancement  quotas  and  exam  cutting  scores  to  the 
Series  51  data. 
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2.  Alternatives  Investigated 


For  advancement  to  E-4/5/6/7,  six  components  are  weighted  as  indi¬ 
cated  in  Table  1  (see  Appendix  A  for  all  Tables  and  Figures).  The 
composites,  i.e..  Final  Multiple  Score  (FMS),  of  competing  candidates 
are  rank-ordered  within  each  Rate  (i.e.,  within  technical  specialty 
and  pay  grade).  Advancements  are  effected  from  the  top  of  the  rank- 
ordered  list  of  composite  scores.  However,  candidates  failing  to  score 
above  a  certain  cut-off  point  on  the  examination  are  not  considered 
further  irrespective  of  their  FMS.  These  cutting  scores  vary  widely, 
from  as  low  as  the  4th  percentile  in  Pay  Grade  4  in  some  Ratings,  to 
as  high  as  the  66th  percentile  in  Pay  Grade  7  in  other  Ratings.  There 
are  thus  three  possible  outcomes--Fail  the  examination  (F) ,  Pass  the 
examination  and  be  Advanced  (A) ,  and  Pass  the  examination  but  Not  be 
Advanced  (PNA).  The  outcomes  from  five  alternative  weighting  designs 
(the  operational  design  and  four  experimental  variations)  are  presented 
in  this  report: 

a.  Operational  -  for  Series  42,  51,  and  simulated  55  data 
(Method  1) . 

b.  Experimental  -  for  a  few  representative  Ratings  of  each  pay 
grade  from  simulated  Series  55  data,  present  operational  procedures 
(Method  1)  were  modified  as  follows:  (Components  are  the  same  as  those 
indicated  in  Table  1,  although  Good  Conduct  Medals  and  Other  Awards  are 
combined  in  one  "Awards"  (AWD)  component.  NC  indicates  No  Change.  The 
new  component,  PNA,  represents  two  points  per  previous  PNA.) 


Method  Alternative  Maximum  Points 


EM 

HR 

LOS 

FMS 

1 

Operational 

-- 

20 

20 

185 

2 

PNA  (20) 

20 

NC 

NC 

205 

3 

PNA  (TIR  10) 

10 

10 

NC 

NC 

4 

PNA  (LOS  10) 

10 

NC 

10 

NC 

5  LOWCUTS  -  No  Change  from  points  indicated  in  Table  1, 

but  lower  cutting  scores  (see  Table  2)  than 
present  operational  procedures. 

c.  For  an  indication  of  what  enlisted  personnel  think  the  propor¬ 
tionate  influences  are  and  should  be,  two  unstructured  and  five  struc¬ 
tured  items  were  designed  (see  Appendix  C)  and  administered  to  a  small 
sample  of  personnel  in  Pay  Grades  4-7. 

d.  With  the  exception  of  the  PNA  Bonus,  only  components  presently 
in  the  Advancement  Multiple  (FMS)  were  investigated. 

3.  Analysis 

a.  Criterion.  Using  multiple  regression  analysis,  _r8  products 
were  computed  as  the  measure  of  proportionate  influence  for  each 
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component.  Employing  the  methodology  discussed  in  Appendix  D  and 
demonstrated  in  Table  3,  the  dichotomous  criterion  of  Advanced-Not 
Advanced  was  compared  with  the  composite  score  (FMS). 

b.  Longitudinal  Incidence  of  PNA  and  Exam-Fail.  Taking  the  most 
recent  set  of  advancement  data  as  the  base  (Series  51)  the  number  of 
candidates  in  each  Rate  group  who  PNA'd  or  failed  the  exam  once,  twice, 
etc. ,  up  to  six  times,  in  the  previous  three  years,  was  tallied. 

c.  Comparison  of  Selectees.  Characteristics  of  candidates  se¬ 
lected  by  each  alternative  method  were  computed  for  two  categories  of 
selected  candidates: 

(1)  For  the  candidates  selected  uniquely  (i.e.,  not  the  entire 
advanced  group)  the  mean  of  each  component  was  obtained  for  each  ex¬ 
perimental  procedure  and  compared  with  the  mean  for  the  operational 
procedure. 

(2)  The  proportion  of  the  entire  advanced  group  drawn  from 
each  quarter  of  each  component  was  analyzed  by  a  computer  program, 
the  Tetra  Summary,  developed  for  this  purpose.  Each  component  was 
independently  rank-ordered,  partitioned  into  quarters,  and  the  propor¬ 
tion  advanced  from  each  quarter  tallied. 

Only  a  few  representative  Ratings  of  each  pay  grade,  those 
with  PNA  in  Series  51,  were  included  in  the  in-depth  analyses  described 
above.  These  were: 


Pay  Grade 


Ratings 


4  ABH,  PN 

5  ABH,  FTG,  PN 

6  ABH,  AT,  FTG,  PN 

7  ABH,  AT,  FTG,  PN 


d.  Survey  Analysis.  Means,  stmdard  deviations,  and  frequency 
distributions  were  computed  for  the  items  on  the  questionnaire. 
Appendix  C,  asking  what  the  proportionate  influences  should  be. 


C.  RESULTS 


1.  Previous  PNA  and  Fails 


As  indicated  in  Table  4,  the  incidence  of  previous  Fails  is  much 
greater  than  of  previous  PNA.  The  highest  incidence  of  both  (exclud¬ 
ing  the  Petty  Officer  third  class  Rates,  in  which  advancement  oppor¬ 
tunity  is  seldom  restricted  and  exams  are  sometimes  administered 
quarterly)  is  for  competition  to  ABHC,  in  which  24  percent  of  the 
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candidates  had  one  or  two  previous  PNA,  and  42  percent  had  one  or  two 
previous  Fails.  Generally,  it  may  also  be  noted  that: 

a.  Many  candidates  have  five  or  more  Fails. 

b.  Few  candidates  have  more  than  three  PNA,  and 

c.  Incidence  of  Fails  is  fairly  consistently  high  across  Ratings 
as  compared  with  PNA's,  which  vary  from  zero  to  many,  depending  upon 
advancement  opportunity. 

2.  Proportionate  Influences  of  Components 

a.  Table  5a  permits  a  comparison  of  the  effective  (as  defined  in 
Appendix  D  and  distinguished  from  nominal)  proportionate  weights  of 
the  components  when  applying  a  dichotomous  (Advanced-Not  Advanced)  and 
a  continuous  (FMS)  criterion  variable.  As  stated  above,  the  February 
1967  data  reflect  a  period  of  relatively  high  active  duty  force  levels 
It  may  be  noted  that: 

(1)  Considering  the  median  weights  of  the  above  two  types  of 
variables  on  the  first  two  components  indicated  in  Table  5a, 


EXAM  PERF 


Percent  PNA 

A/NA 

FMS 

A/NA 

FMS 

None 

99 

67 

0 

23 

1-10 

99.5 

63.5 

1.5 

23 

More  than  11 

96 

61 

3 

19 

when  the  dichotomous  criterion  (A/NA)  is  applied,  it  indicates  that 
the  exam  is  virtually  the  sole  determinant  of  advancement,  especially 
with  few  or  no  PNA;  while  the  continuous  criterion  (FMS)  permits  the 
exam  to  contribute  much  less  weight. 

(2)  If  all  of  those  passing  an  exam  are  advanced,  and  all  of 
those  failing  the  exam  are  not  advanced  (i.e.,  no  PNA),  the  exam  obvi¬ 
ously  must  be  the  sole  determinant  of  advancement.  Yet,  the  range  of 
exam  weights  in  Table  5a,  applying  the  FMS  as  the  criterion  variable, 
for  the  19  Rate  groups  with  no  PNA,  is  noticeably  less  than  100  per¬ 
cent  weight,  i.e.,  42-77  percent. 

(3)  Some  of  the  weights  are  slightly  negative  or  in  excess  of 
100  percent,  for  the  reasons  stated  in  Appendix  D. 

b.  The  Rate  groups  in  Table  5b,  from  data  which  reflect  a  pexiod 
(1969-1970)  after  substantial  reduction  in  force  levels,  are  arranged 
by  pay  grade,  rather  than  by  PNA  incidence  as  in  Table  5a.  (The 
reason  for  the  pay  grade  grouping  was  to  avoid  an  implication  of  a 


higher-than-actual  PNA  base  rate  among  Rate  groups,  since  many  high- 
PNA  Rates  had  intentionally  been  selected  for  the  purpose  of  studying 
that  particular  problem.)  Table  6  presents  the  range  and  median  of 
the  simulated  August  1970  weights  from  Table  Sb.  It  may  be  noted 
that : 


(1)  Again,  the  exam  weights,  applying  the  continuous  (FMS) 
criterion,  are  noticeably  lower  than  when  applying  the  dichotomous 
criterion. 

(2)  On  each  component,  there  are  extreme  differences  in  weights 
across  both  Rating  and  pay  grade  groups. 

(3)  Although  the  simulated  Series  55  data  indicate  slightly 
less  dominance  by  the  exam  component  than  the  Series  42  data  (in  part 
because  of  the  intentional  selection  of  high  PNA  Rates),  the  exam  still 
accounts  for  greater  than  90  percent  of  the  weight  in:  Two  of  the  four 
Pay  Grade  4  Ratings,  five  of  eight  Pay  Grade  5,  four  of  thirteen  Pay 
Grade  6,  and  five  of  fifteen  Pay  Grade  7  Ratings. 

3.  Expressed  Preference  for  Proportionate  Weights 

a.  Figures  1-6  display  the  distribution  of  responses  to  Questions 
1-7,  Appendix  C.  Questions  1  and  2  (what  the  proportionate  influences 
actually  are  and  should  be.  Figures  1  and  2,  respectively)  were  open- 
ended.  The  enlisted  respondents  could  express  their  opinion  by  enter¬ 
ing  any  combination  of  values  totaling  100  percent.  There  was  fairly 
close  agreement  among  all  pay  grades  both  on  "actual"  and  "should  be" 
opinions.  Generally,  however,  the  respondents  indicated  that  exam  and 
experience  should  be  reduced  slightly,  and  performance  increased 
slightly  over  what  they  thought  the  weights  actually  were.  Taking  the 
Pay  Grade  7  "should  be"  responses  as  an  approximate  median,  the  pro¬ 
portions  are: 


EXAM 

PERF 

LOS 

TIR 

Good 

Cond. 

Other 

AWD 

Any 

Other 

54 

26 

5 

6 

3 

2 

2 

b.  Questions  3-7  were  structured  to  investigate  the  differential 
preference  for  a  wide  variety  of  sets  of  proportionate  influences. 
Generally,  those  combinations  high  on  performance,  both  on  technical 
knowledge  (as  measured  by  the  exam)  and  on- job  (as  measured  by  evalua¬ 
tion  marks),  were  considered  the  fairest.  As  Figure  3  indicates,  the 
combinations  presented  in  Questions  5,  7  and  4  were  considered  the 
fairest;  Questions  6  and  3,  the  least  fair. 

c.  Although  there  was  close  agreement  on  the  mean  fairness  among 
all  four  pay  grades,  there  was  substantial  disagreement  on  the  degree 
of  fairness  within  pay  grade.  The  high:-:t  (i.e.,  modal)  frequency  of 
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a  particular  degree  of  fairness,  Questions  3-7  on  Figure  4,  was  only 
40  percent  (for  Question  5).  Consider,  for  example,  the  distribution 
of  fairness  responses  for  Question  4,  in  which  the  modal  response  of 
29  percent  is  £--"slightly  fair."  However,  the  "unfair"  responses 
(d,  e  and  f)  total  45  percent. 

d.  Figure  5  displays  the  distribution  of  fairness  responses  by 
pay  grade  for  Question  5,  which  was  highest  among  Questions  3-7  on 
fairness,  and  which  also  most  closely  approximates  the  intended  com¬ 
ponent  weights  as  represented  to  the  Fleet  (Table  1). 

4.  Effect  of  Alternative  Weighting  Procedures 

a.  In  analyzing  the  outcomes  of  the  five  alternative  methods  (as 
defined  in  above  paragraphs  B.2.a.  and  b.),  it  is  useful  to  consider 
each  Rate  group  as  falling  within  one  of  four  PNA  categories.  The 
thirteen  Rate  groups  (a  few  from  each  pay  grade)  selected  fcr  this 
part  of  the  analysis  were  categorized  as  follows: 

Percentage  of  Candidates  with  Incidence  of  PNA 
Present  PNA 


High  Low 

(17-38%)  (0-6%) 


Incidence  Hi8h 
of  (6-35%) 

PN1,  ABHC, 
PNC,  ATC 

ABH3,  PN3, 
PN2 

Previous 

PNA's  ^ow 

(0-1%) 

ABH1,  FTGC 

ABH2,  FTG2, 

FTG1,  ATI 

Another  consideration  is  the  existence  of  a  skewed  distribu¬ 
tion  on  any  component  to  which  a  new  maximum  or  minimum  is  imposed  by 
an  alternative  method  which  appreciably  curtails  the  distribution. 

For  example,  the  10-point  maximum  PNA  bonus  of  Method  3  affects  no 
Rate  group,  since  there  are  virtually  no  cases  of  more  than  four  pre¬ 
vious  PNA.  However,  some  Rates  have  a  substantial  pile-up  of  TIR 
points  so  that  the  TIR  maximum  of  10  points  of  Method  3  makes  a  sub¬ 
stantial  difference. 

b.  Given  the  above  considerations,  the  following  observations 
may  be  made  from  Table  7: 

(1)  Regardless  of  previous  PNA,  with  little  or  no  present  PNA, 
the  PNA  Bonus  Methods  (2,  3  and  4)  have  little  or  no  effect  which  is 
different  from  the  operational  method  (e.g.,  ABH3,  PN3,  ABH2,  FTG2,  PN2, 
ATI,  and  FTG1) . 
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(2)  With  a  high  present  PNA,  but  little  or  no  previous  PNA, 
there  is  little  difference  in  effect  (e.g.,  FTGC) ,  unless  the  distri¬ 
bution  of  a  component  is  so  high  that  it  is  substantially  curtailed 
by  a  newly  imposed  maximum  (e.g.,  for  ABH1,  no  PNA  effect,  but  lower 
LOS  by  Method  4) . 

(3)  With  both  present  and  previous  high  PNA,  the  PNA  bonus 
methods  generally  increase  the  exam  influence  (from  the  combined 
weights  of  the  exam  component  and  the  PNA  component,  e.g.,  applying 
Method  3  to  ABHC,  84.9  +  9.61;  lower  LOS,  TIR  and  Awards;  and  do  not 
change  the  performance  component  (e.g.,  PN1,  ABHC,  ATC,  and  PNC). 

(4)  Although  there  are  some  exceptions,  reducing  the  exam  cut- 
scores  (Method  5)  generally  decreases  the  exam  weight  and  increases 
the  other  component  weights  (e.g.,  ABH2,  FTG2,  PN2,  ATI  and  FTG1) . 

The  changes  are  less  extensive  and  more  variable  in  those  Rate  groups 
with  a  present  high  PNA,  because  the  exam  cut-score  is  not  at  the  ad¬ 
vancement  opportunity,  thereby  enabling  the  other  components  to  con¬ 
tribute  some  weight. 

c.  Since  the  primary  interest  in  the  direction  and  extent  of 
change  in  weights  by  alternative  methods  should  be  in  relation  to  some 
objective.  Table  8  was  constructed  by  applying  the  data  of  Table  7  to 
a  specific  criterion.  (As  stated  in  Table  8,  the  criterion  used  was 
an  approximation  of  the  weights  favored  by  the  enlisted  respondents 

to  the  Questionnaire  in  Appendix  C.)  In  terms  of  convergence  towards, 
or  divergence  from  the  criterion,  Table  8  Method  Summaries  indicate, 
from  the : 

(1)  PNA  bonus  methods,  divergence  on  LOS,  and  for  Method  4 
only,  convergence  on  performance.  (However,  when  the  PNA  bonus  is 
considered  as  part  of  the  exam  component,  the  combined  effect  is  a 
slight  divergence.) 

(2)  Lower  Cuts  method,  convergence  on  exam  and  performance, 
and  divergence  on  TIR. 

All  other  effects  were  slight  or  variable.  As  stated  above, 
the  extent  of  the  effect  is  dependent  upon  the  incidence  of  present 
PNA- -decreasing  the  relative  effect  of  the  Lower  Cuts  method;  and  in¬ 
creasing  (if  also  previous  PNA)  the  PNA  bonus  effect. 

d.  Would  different  candidates  actually  be  selected  by  one  weight¬ 
ing  method  in  lieu  of  another?  If  so,  to  what  extent?  And  what  are 
the  qualitative  differences  between  those  uniquely  selected  and  the 
other  candidates  consequently  not  selected  (i.e.,  "bumped")? 

(1)  From  Table  9,  it  is  evident  that  the  PNA  bonus  methods 
could  select  substantial  proportions  of  the  advanced  group  uniquely 
different  from  the  present  operational  method  (e.g.,  8-57  percent), 
dependent  upon  both  present  and  previous  high  PNA  (PN1,  ABHC,  ATC  and 
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PNC).  (Thus,  since  ABH2,  FTG2,  FTG1,  FTUC  and  ATI  had  only  one  or  no 
previous  PNA' s,  the  PNA  bonus  points  had  no  effect.)  Although  the 
proportion  of  uniquely  selected  candidates  is  large  for  some  methods, 
the  number  of  candidates  is  usually  small  because  of  the  extremely 
restricted  advancement  opportunity.  By  the  Lower  Cuts  method,  the 
lower  the  present  or  previous  PNA  incidence,  the  greater  is  the  pro¬ 
portion  of  unique  selections  (e.g.,  ATI  and  FTGC) . 

(2)  The  qualitative  mean  differences1  presented  in  Table  10 

2 

generally  reflect  the  same  shifts  as  indicated  in  Table  8.  Consider¬ 
ing  again  the  PNA  categories  of  above  paragraph  C.4.a.,  when  candidates 
are  uniquely  selected  by  a  given  experimental  method,  it  may  be  ob¬ 
served  that: 


(a)  With  present  low  PNA,  regardless  of  previous  PNA  (Rate 
groups  1N2,  ABH2,  FTG2,  FTG1  and  ATI),  only  Method  5  (Low  Cuts)  re¬ 
sulted  in  usually  significant  differences  of  lower  exam  score)  better 
on-job  performance  and  awards,  and  greater  experience  (LOS  and  TIR). 
(The  only  Rate  groups  without  differences,  ABH3  and  PN3,  had  very  high 
advancement  opportunity,  90  and  94  percent  respectively.  Thus,  cut- 
scores  were  already  nearly  as  low  as  the  experimental  Method  5  cuts, 
and  no  candidates  were  uniquely  selected.) 

(b)  Combining  Rates  with  both  present  and  previous  high 
PNA  (PN1,  PNC,  ABHC,  ATC) : 

i.  Method  2  had  no  significant  effect. 

ii.  Methods  3  and  4  raised  exam  and  performance  means 
(Method  4  only)  and  lowered  LOS,  TIR  and  Awards. 

iii.  Method  5  results  were  similar  to  the  above  low  PNA 
groups,  but  differences  in  performance  were  not  quite  significant 
(P  <  •  06) . 


*ln  analyzing  qualitative  differences  uong  only  the  uniquely  selected  candidates,  especially  in  the  Pates  with 
extremely  limited  advancement  opportunity,  the  ver*  small  N’s  pose  a  severe  problem  for  significance  tests.  Is  each 
Rate  group  to  be  viewed  as  a  total  population  or  as  a  saapTe  of  a  recurring  situation  over  several  advancement  com¬ 
petitions?  Some  arguments  which  support  the  population  view  are:  The  present  dat.  set  comprise  a  PNA  history  ex¬ 
tending  over  several  exam  series  and  years;  and  advancement  opportunity  and  force  levels  vary  over  time.  Thus  the 
combining  of  total  Rate  groups  across  exam  series  may  not  be  Mssxlples,,  from  the  same  population.  (However,  since 
one  of  the  principal  quest2 * 4ons  of  .merest  in  the  present  study  is  the  nature  of  the  PNA  problem,  some  Rate  groups 
with  a  common  characteristic-^  high  incidence  of  both  present  and  previous  PNA,  were  combined  in  order  to  Increase 
sample  site,  and  are  also  presented  in  Table  10.)  It  would  be  no  consolation  to  a  few  candidates  (but  who  comprise 
a  substantial  proportion  of  the  advanced  group)  to  be  advised  that  the  differences  between  their  qualifications 
(hi rein  component  scores)  and  those  of  a  few  others  selected  in  lieu  of  them  by  an  alternative  method,  were  large 
but  not  statistically  significant.  Thus  the  data  presented  for  small  N's,  if  not  supportive  are  certainly  strongly 
suggestive  of  differential  effects  from  alternative  weighting  systems. 

2As  stated  in  Note  a  of  Table  10,  the  means  were  computed  from  the  actual  component  values  (i.e.,  to  reflect  the 

actual  characteristics  of  the  candidates),  although  the  maximum  cut-offs  were  applied  in  making  the  selections.  For 
example,  applying  Method  3  to  the  ATC  Rate,  which  resulted  in  three  unique  selections,  the  mean  TIR  of  the  three 
"bunded"  candidates--123.66,  was  coajwted  from  scores  of  126,  149  and  96.  However,  the  selections  by  Methods  1,  2, 

4  and  5  were  based  on  scores  of  120,  P0  and  96  respectively;  and  by  Method  3,  of  60,  60  and  60. 
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S.  Discussion 


a.  Although  the  above  comparison  of  uniquely  selected  candidates 
provides  useful  insights  concerning  differences  among  alternative 
weighting  methods,  the  ultimate  concern  is  with  the  qualitative  char¬ 
acteristics  of  the  total  groups  selected  by  a  particular  method. 

Table  11  presents  a  sample  of  the  computer- generated  Tetra  Summary. 
Table  12  shows  the  proportion  advanced  within  each  quarter  for  the 
thirteen  Rate  groups  studied. 

b.  Presumably,  a  multi-component  system  should  select  the  "best 
performers,"  in  terms  of  technical  knowledge  and  on- job  behavior, 
from  the  high  end  of  each  appreciably  weighted  component.  This  would 
be  reflected  by  consistent  decreases  in  proportions  advanced  from  the 
upper  through  the  lower  segments  of  each  component's  distribution. 
Table  12  indicates  this  always  to  be  the  case,  by  Method  1,  for  the 
exam  component,  Lut  seldom  the  case,  for  performance.  Some  quarter 
other  than  the  top  quarter  of  on- job  performance  usually  contains  the 
highest  •  roportion  of  advancements.  In  one  case,  the  bottom  quarter 
(FTG1);  in  another  case,  all  other  quarters  are  higher  than  the  top 
quarter  (ATI) . 

c.  Figures  7  and  8  display  the  resulting  shifts  in  slope  from 
the  alternative  weighting  methods  for  cases  representative  of  various 
PNA  incidence.  It  may  be  noted  that: 

(1)  The  higher  the  present  PNA,  the  more  the  other  components 
(besides  exam)  tend  towards  a  downward  slope,  and  the  less  relative 
shift  in  slope  from  Method  5  (Lower  Cuts). 

(2)  If  low  present  or  previous  PNA,  no  shift  results  from  any 
PNA  bonus  (Method  2,  3  or  4) . 

(3)  If  no  present  PNA,  ascending  slopes  can  occur  on  other 
than  exam  components,  as  is  strikingly  the  case  with  ATI.  Lower  exam 
cuts  reverse  the  slope. 


D.  SUMMARY  OF  FINDINGS 

The  relevancy  of  alternative  methods  for  analyzing  the  Enlisted 
Advancement  System  were  investigated.  The  outcomes  from  five  alterna¬ 
tive  weighting  systems  as  applied  to  the  components  in  the  advancement 
composite  were  also  demonstrated. 

1.  The  outcomes  from  a  dichotomous  (advanced  vs.  not  advanced)  and  a 
continuous  (FMS)  dependent  variable  were  compared,  using  multiple- 
regression  analysis.  The  dichotomous  criterion  was  found  to  be  the 
more  relevant,  because  a  continuous  criterion  is  not  sensitive  to  the 
influence  of  component  cut-off  scores  and  advancement  quotas. 
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2.  When  wide  fluctuations  in  component  distributions  occur  across 
and  within  pay  grades  and  Ratings,  they  produce  similar  wide  fluctu¬ 
ations  in  the  effective  (as  defined  in  Appendix  D)  weights  of  the 
components. 

3.  Differences  in  exam  cut-off  points  and  advancement  quotas  (also 
across  pay  grades  and  Ratings)  cause  the  proximity  of  cut  score  to 
selection  opportunity  to  have  a  substantial  and  varying  influence 

on  advancement.  There  is  a  much  greater  incidence  of  exam-fail  than 
PNA.  Thus  it  is  the  paucity,  rather  than  the  high  prevalence,  of 
PNA  which  poses  the  greater  problem. 

4.  A  newly  developed  nonparametric  method,  the  Tetra  Summary,  was 
found  to  be  very  useful  in  comparing  the  quality  levels  among  com¬ 
ponents  for  the  advanced  group. 

5.  Enlisted  personnel  indicate  a  preference  for  proportionate 
weights  of  the  components  fairly  close  to  the  published  weights. 

6.  Alternative  weighting  systems  of: 
a.  PNA  bonus  points 

(1)  Increase  further  the  exam  weight  at  the  expense  of  LOS 

and  TIR. 


(2)  Select  about  25  to  50  percent  of  the  advanced  candidates 
uniquely  different  from  the  operational  system,  only  if  there  exists 
for  that  Rate  group  a  high  incidence  of  both  present  and  previous  PNA. 

b.  Lower  exam  cut -points  achieve  effective  component  weights 
which  tend  toward  the  nominal  (i.e.,  the  intended  or  policy)  weights, 
thereby  decreasing  the  exam  weight  and  raising  the  weights  of  other 
components. 


E.  RECOMMENDATIONS 


1.  Short-Term  Recommendations 


a.  If  it  is  desired  that  effective  weights  more  closely  approxi¬ 
mate  nominal  weights,  apply  lower  exam  cut-scores  for  competition  to 
Pay  Grades  5,  6  and  7. 

b.  If  it  is  desired  to  reward  previous  high  exam  scores  only, 
with  the  present  system  otherwise  continuing  to  function  the  same, 

(1)  Apply  the  PNA  bonus  as  an  addition  to  the  composite  (FMS), 
i.e.,  Method  2,  although  a  maximum  of  10  points,  vice  20  points,  would 
suffice. 
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(2)  Do  not  apply  a  PNA  bonus  in  lieu  of  TIR  or  LOS,  since  this 
could  increase  exam  weight  and  reduce  LOS  and  TIR  weight  even  in  cases 
of  no  previous  PNA  incidence. 

2.  Long-Term  Recommendations 

The  purpose  of  this  Technical  Bulletin  was  limited  to  the  demon¬ 
stration  of  outcomes  from  comparison  of  alternative  weighting  and 
analytical  methods.  No  major  changes  to  the  present  operational  system 
are  recommended  at  this  time.  Prior  to  making  such  recommendations, 
research  is  needed  to: 

a.  Determine  the  differential  relevance  (across  Ratings  and  pay 
grades)  of  the  various  performance  factors  (on-job,  technical  knowl¬ 
edge,  and  awards)  to  qualifications  for  the  next  higher  Rate  and  to 
retention. 

b.  Develop  an  adaptive  advancement  system,  employing  iterative 
computer  programs  capable  of  achieving  policy  weights  given  substan¬ 
tial  fluctuations  in  component  distributions  and  advancement  opportun¬ 
ity,  both  among  and  within  pay  grades  and  Ratings. 
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TABLE  1 


Components  and  Maximum/Minimum  Points  in  the  Final  Multiple 
Score  for  Advancement  in  Rating  to  Pay  Grades  E-4/5/6/7 


Component 

Maximum 

Points 

Maximum  as 
Percentage  of 
Composite  (FMS) 

Multiplier 

Minimum 

Points 

Examination  Score 

80 

43.3 

1 

Varies8 

Performance  Evaluations 

50 

27.0 

30P-70 

(None) 

Length  of  Service  (LOS) 

20 

10.8 

1  X  Months 
12 

(None) 

Service  in  Pay  Grade  (Time 
in  Rate--TIR) 

20 

10.8 

2  X  Months 

12 

(None) 

Good  Conduct  Medals 

10 

5.4 

1 

(None) 

Other  Awards 

5 

2.7 

1 

(None) 

Final  Multiple  Score  (FMS) 

185 

100.0 

Note. --Weighting  factors  listed  are  "nominal"--or  intended  weights.  In 
actual  practice  the  effects  of  the  cuts  used  yield  results  highly  discrep¬ 
ant  from  those  desired.  See  text  for  discussion. 


Minimum  (i.e.,  "cutting-score")  varies  by  pay  grade  from  standardized 
"T"  scores  (X  =  50,  SD  *  10)  of  32  for  some  E-4  Ratings,  to  54  for  some 
E-7  Ratings.  See  Table  2. 
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Advancement  Opportunity  and  Exam  Cut  Point  by  Pay  Grade  for  Two  Operational 
(Series  42  and  55) ,  and  One  Hypothetical  Advancement  Series 
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Excludes  extremely  atypical  case  of  MNC  of  44  percent  advancement  opportunity. 


Intercorrelations  and  Correlation-Beta  Weight  Products  of  Advancement  Components 
For  the  Personnelman  (PN)  Rating,  Pay  Grades  E-4/5/6/7  (Series  51} 
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TABLE  4 


Incidence  of  Previous  PNA  and  Exam-Fail  of  Series  51  (August  1969) 
Candidates  During  Calendar  Years  1967-1969  Series 


Series 

51 

Frequency  of  Previous: 

Candidates 

PNA 

Exam- 

Fail 

First 

Rate 

Total 

1 

2 

3 

4  or 

1 

2 

3 

4 

5  or 

Try 

more 

more 

ABH3 

N 

418 

113 

33 

5 

. 

105 

55 

22 

8 

1 

136 

% 

27 

8 

1 

- 

25 

13 

5 

2 

0.2 

33 

PN3 

N 

605 

119 

46 

_ 

- 

154 

80 

17 

5 

5 

221 

% 

- 

20 

8 

- 

- 

25 

13 

3 

1 

1 

37 

ABH2 

N 

265 

- 

- 

- 

- 

12 

2 

3 

2 

5 

241 

% 

- 

- 

- 

5 

1 

1 

1 

2 

91 

FTG2 

N 

436 

- 

- 

- 

- 

47 

20 

5 

2 

- 

359 

% 

- 

- 

- 

“ 

11 

5 

1 

0.5 

“ 

82 

PN2 

N 

911 

189 

2 

- 

- 

211 

63 

17 

9 

.  - 

448 

% 

- 

21 

0.2 

- 

- 

23 

7 

2 

1 

- 

49 

ABH1 

N 

176 

2 

- 

- 

- 

34 

24 

14 

10 

10 

83 

% 

1 

- 

- 

19 

14 

8 

6 

6 

47 

ATI 

N 

563 

1 

- 

- 

m 

328 

% 

- 

0.2 

- 

- 

- 

MA 

■fl 

58 

FTG1 

N 

141 

- 

- 

- 

- 

25 

SI 

E 

7 

- 

72 

% 

- 

- 

- 

- 

- 

18 

5 

- 

51 

PN1 

N 

% 

519 

23 

4 

9 

2 

- 

- 

m 

Eh 

H 

258 

50 

ABHC 

N 

230 

45 

9 

2 

- 

59 

37 

26 

El 

•ax 

60 

% 

- 

20 

4 

1 

- 

26 

16 

11 

m  -  mm 

26 

ATC 

N 

749 

105 

33 

6 

2 

Wr 

279 

% 

- 

14 

4 

1 

0.3 

El 

■rat-.. 

37 

FTGC 

N 

134 

- 

- 

- 

- 

33 

12 

11 

- 

- 

78 

% 

“ 

* 
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25 

9 

8 
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- 

58 

PNC 

N 
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Comparison  of  a  Dichotomous  (Advanced-Not  Advanced — A/NA)  and  a  Continuous 
(Final  toiltiple  Score--FMS)  Criterion  on  Proportionate  Influence  of 
Components  by  Incidence  of  PNA  (Series  42,  February  1967,  Data) 
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.rtULE  5a  (Continued) 
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Comparison  of  a  Dichotomous  (Advanced-Not  Advanced- -A51,  A55)  and  a  Continuous  (Final 
Miltiple  Score--F51)  Criterion  on  Proportionate  Influence  of  Components  (Series 
51,  August  1969;  and  Simulated  Series  55,  August  1970  Data) 

Criteria:  F51--FMS,  Series  51 
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TABLE  5b  (Continued) 
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Differences  Among  Ratings  and  Pay  Grades  in  Proportionate  Influence 
Of  Components  (Simulated  Series  55,  August  1970,  Data) 
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TABLE  7 


Proportionite  Influences  of  Components  Among  the  Operational 
(Method  1)  and  Pour  Experimental  (Methods  2-5)  Weighting 
Procedures  (Simulated  Series  55,  August  1970,  Data) 


Rate 

N 

Method 

Component 

Percentage 

EXAM 

PERF 

LOS 

TIR 

AND 

PNA* 

BONUS 

Series  55 
Advanced/ 
PNA/Fai  1 
(A/l’/F) 

Of  Previous  PNA's 

No. 

1 

of  PNA's: 

2  3  4 

m 

'BP 

bh 

xa 

«J 

0.7 

NA 

2 

mm 

m 

1.1 

2.1 

ABH3 

412 

3 

BH 

1.2 

2.1 

90/06/04 

27 

8 

1 

4 

♦fin 

1.2 

2.1 

5 

MSi 

0.7 

NA 

1 

96.3 

1.9 

i.i 

1.5 

-0.7 

NA 

2 

95.9 

1.8 

i.i 

1.4 

-0.7 

0.4 

PN3 

579 

3 

95.9 

1.8 

i.i 

1.4 

-0.7 

0.4 

94/00/06 

20 

8 

- 

4 

95.9 

1.8 

i.i 

1.4 

-0.7 

0.4 

5 

96.3 

1.9 

i.i 

1.5 

-0.7 

NA 

1 

93.4 

1.4 

1.9 

-0.2 

3.5 

NA 

2 

93.4 

1.4 

1.9 

-0.2 

3.5 

— 

ABH2 

265 

3 

93.4 

1.4 

1.9 

-0.2 

3.5 

— 

72/00/28 

- 

- 

- 

4 

93.4 

1.4 

1.9 

-0.2 

3.5 

— 

5 

90.9 

6.7 

-0.4 

-1.1 

3.8 

NA 

1 

92.0 

7.8 

-7.2 

7.2 

0.2 

NA 

2 

92.0 

7.8 

-7.2 

7.2 

0.2 

— 

ETC  2 

433 

3 

92.0 

7.8 

-7.2 

7.2 

0.2 

— 

72/00/28 

- 

- 

- 

4 

92.0 

7.8 

-7.2 

7.2 

0.2 

— 

5 

73.3 

28.5 

-2.1 

... 

0.2 

NA 

1 

95.8 

4.8 

-0.6 

-0.2 

NA 

2 

EH 

4.8 

-0.6 

— 

— 

3.8 

PN2 

903 

3 

4.8 

-0.6 

... 

— 

3.8 

72/00/28 

21 

- 

4 

92.0 

4.8 

-0.6 

— 

— 

3.8 

5 

84.3 

16.1 

— 

-0.4 

... 

NA 

1 

76.7 

3.4 

2.2 

0.4 

17.2 

NA 

2 

76.3 

3.4 

2.1 

0.4 

17.6 

— 

ABH1 

175 

3 

76.3 

3.4 

2.1 

0.4 

17.6 

--- 

04/38/58 

1 

- 

- 

4 

87.9 

3.0 

— 

1.3 

7.8 

— 

5 

76.7 

3.4 

2.2 

0.4 

17.2 

NA 

1 

99.1 

_ 

1.8 

0.2 

-1.1 

NA 

2 

98.9 

— 

1.8 

0.2 

-1.1 

0.2 

ATI 

561 

3 

98.9 

— 

1.8 

0.2 

-1.1 

0.2 

40/00/60 

- 

- 

- 

4 

98.9 

— 

1.8 

0.2 

-1.1 

0.2 

5 

89.0 

3.7 

2.1 

2.1 

NA 

1 

96.2 

-1.2 

-0.3 

NA 

2 

96.2 

n 

-1.2 

-0.3 

— 

FTG1 

141 

3 

96.2 

0.2 

-1.2 

-0.3 

--- 

48/00/52 

- 

- 

- 

4 

96.2 

0.2 

-1.2 

-0.3 

— 

5 

91.7 

7.6 

-1.5 

2.4 

-0.2 

NA 

1 

42.4 

2.9 

3.8 

50.0 

0.9 

NA 

2 

28.4 

3.2 

4.0 

60.4 

-0.9 

5.0 

PN1 

519 

3 

72.3 

5.6 

5.6 

4.5 

9.0 

2.8 

04/38/58 

4 

2 

- 

4 

50.0 

3.7 

0.9 

46.3 

— 

-0.5 

5 

26.5 

2.9 

5.5 

67.2 

-2.1 

NA 

(Continued  on  next  page) 
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TABLE  7  (Continued) 


Component  _ Percentege _ 

- -  Series  5S  Of  Previous  PNA's 


Rate 

N 

He  t  hex 

EXAM 

PERF 

LOS 

TIR 

AMD 

PNA* 

BONUS 

Advanced/ 

PNA/Fail 

(A/P/F) 

No. 

1 

of  PNA's: 

2  3  4 

i 

S3. 6 

-0.6 

S.4 

23.8 

17.3 

NA 

2 

67.3 

-0.6 

2.4 

14.9 

4.8 

10.7 

ABHC 

206 

3 

84.9 

— 

— 

4.2 

1.8 

9.6 

03/28/67 

20 

4  1  - 

4 

S9.6 

1.4 

2.4 

1.0 

— 

3S.1 

S 

S3. 6 

-0.6 

5.4 

23.8 

17.3 

NA 

*Not  Applicable  (NA)  since  ro  PNA  Bonus  was  applied  to  Method  1  or  S. 
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TABLE  8 


Relative  Shift  in  ifi  Weights  From  the  Opersttonal  to  Alternative 
Weighting  Methodi  (Data  Source,  Table  7) 


Direction  of  Shift*  by  Alternative  Method 

«  -  42  PNA  20  >  -  #4  PNA  10  (LOS  10) 

4  -  »3  PNA  10  (T1R  10)  /  •  »S  Low  Cuts 


Component  (and  Criterion  Percentage) 


Advance/ 

EXAM 

PERF 

LOS 

TIR 

AWD 

Rate 

PNA/ Fail 
(WP/F) 

£551) 

(251) 

(71) 

(51) 

(81) 

NoCh 

TWD 

Away 

NoCh 

TWD 

Away 

NoCh 

TWD 

Away 

NoCh 

TWD 

Away 

NoCh 

TWD 

Away 

ABHJ 

90/06/04 

xA>/ 

^  xA>/ 

A>  / 

X 

,/ 

A> 

/ 

xA> 

PN3 

94/00/06 

»A>/ 

«4>/ 

xA>/ 

xA  >/ 

xA>/ 

ABH2 

72/00/28 

xA>  / 

X^> 

/ 

xA> 

/ 

xA> 

/ 

»4>/ 

FTG2 

72/00/28 

*A> 

/ 

xA> 

/ 

xA> 

/ 

xA> 

/ 

*A>/ 

PN2 

72/00/28 

x£>/ 

*A> 

/ 

xA> 

/ 

xA> 

/ 

xA>/ 

AfiHl 

04/38/58 

.4/ 

> 

*A>/ 

«4/ 

> 

.4/ 

> 

.4/ 

> 

ATI 

40/00/60 

xA> 

/ 

«A> 

/ 

<A>/ 

*A> ' 

j 

*A> 

/ 

FTGl 

48/00/52 

X^> 

/ 

*A> 

✓ 

*A> 

/ 

*A> 

/ 

*A>/ 

PN1 

04/38/58 

> 

»/ 

A> 

X 

A  / 

> 

✓ 

A 

»/ 

A 

«>✓ 

ABHC 

03/28/67 

/ 

xA> 

«✓ 

A> 

J 

*A> 

/ 

xA> 

/ 

xA> 

ATC 

03/28/67 

xA>/ 

«/ 

> 

A 

» 

/ 

A> 

> 

xA/ 

A 

x> 

/ 

FTGC 

12/18/70 

xA> 

/ 

xA> 

j 

xA>/ 

xA> 

/ 

xA> 

/ 

PNC 

15/17/68 

xA>/ 

*A 

>/ 

>/ 

xA 

»4>/ 

/ 

xA> 

Summary : 

Method  Total 

- 

3 

2 

- 

0 

0 

- 

0 

3 

2 

3 

- 

3 

2 

»2  ' 

TWO  minus  Away  1 

0 

-3 

-i 

1 

#3  » 

Total 

4" 

- 

3 

2 

- 

2 

1 

- 

1 

3 

- 

3 

3 

- 

3 

1 

TWD  ainus  Away  1 

1 

-2 

0 

a 

#4  " 

Total 

>»• 

- 

4 

2 

- 

4 

0 

- 

1 

4 

- 

3 

2 

- 

4 

2 

TWD  minus  Away  2 

4 

-3 

1 

2 

#5  " 

Total 

/" 

- 

7 

1 

- 

7 

0 

- 

5 

2 

- 

3 

6 

- 

3 

2 

TWD  minus  Away  6 

7 

_ 

3 

-3 

1 

*The  shift  is  Indicated  in  reference  to  a  change  from  the  operational  weight.  Towards  (TWO,  i.e.,  converging  on). 
Away  fro*  (i.e.,  diverging),  or  No  Change  (NoCh)  froa,  the  criterion  for  that  component.  (The  criteria  used  for  this 
demonstration  were  selected  froa  values  between  the  policy  weights  of  Table  1  and  the  expressed  preference  weights 
of  Figure  2. 


The  extent  of  the 

relative  shift  was  considered  "No  Change" 

(NoCh)  unless  the  shift: 

For  jpB  Size 

Resulted  in  a  relative 
shift  greater  than 

02.00  or  less 

501 

02.01  -  05.00 

251 

OS. 01  -  SO. 00 

101 

50.01  or  greater 

31 

Exa^>le:  Consider  the  shift  in  two  cooj>onents  by  Methods  1 

(operational)  vs  4,  and  1  vs  5  for  the  FTGC  rate 

Method 

CoDoonent:  EXAM 

1  vs  4 

95.2  vs  94.7,  thus  "No  Change"  (NoCh) 

1  vs  S 

88.4  vs  95.2,  a  difference  of  6.8  (greater  than  3%)  "Toward"  the  criterion  of  55  percent 

Coaoonent:  TIR 

1  vs  4 

-1.7  vs  -1.4,  thus  "No  Chang*"  (NoCh). 

-1.7  vs  1.0,  a  difference  of  2.7  (greater  than  501)  "Toward"  the  criterion  of  7  percent. 
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TABLE  9 


Proportion  of  Candidates  Uniquely  Selected  by  Four  Alternative  Experimental 
Weighting  Procedures  in  Comparison  With  the  Present  Operational 
(Method  1)  System  (Simulated  Series  55) 


Proportion  of  Advanced  Candidates 
Uniquely  Selected  by  Experimental 
Rate  Candidates  vs  Operational  Method 

Total  Advanced  Experimental  Methodb  Incidence  of  Previous  PNAa 


2  3  4  S  01234  or 

(1st  more 

try) 


ABH3 

N 

418 

376 

1 

1 

i 

. 

136 

113 

33 

5 

. 

\ 

.3 

.3 

.3 

- 

33 

27 

8 

1 

- 

PN3 

a 

\ 

605 

541 

- 

- 

- 

- 

221 

37 

119 

20 

46 

8 

- 

- 

ABH2 

N 

% 

265 

191 

_ 

9 

241 

- 

- 

- 

5 

91 

- 

- 

- 

- 

FTG2 

N 

% 

436 

314 

- 

- 

- 

22 

7 

359 

82 

- 

- 

- 

- 

PN2 

N 

t 

911 

656 

1 

.2 

2 

.3 

2 

.3 

31 

5 

448 

49 

189 

21 

2 

0.2 

ABH1 

N 

F 

176 

7 

- 

- 

1 

14 

- 

83 

47 

2 

1 

- 

ATI 

N 

563 

225 

- 

_ 

- 

S3 

328 

1 

F 

- 

- 

* 

- 

24 

58 

0.2 

- 

FTG1 

U 

% 

141 

68 

- 

- 

- 

13 

19 

72 

51 

- 

- 

PN1 

N 

\ 

519 

21 

3 

14 

5 

24 

2 

10 

2 

10 

258 

50 

23 

4 

9 

2 

- 

ABHC 

N 

\ 

230 

7 

1 

14 

2 

29 

4 

57 

- 

60 

26 

45 

20 
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TABLE  11 

Co^mter-Generated  Display  of  Proportion  of  Advancements  (A),  PNA--Quota  (Q) ,  and  Exam-Fail  (F) 
Selected  from  Each  Rank-Ordered  Ccxqionent  (garter  by  Alternative  Weighting  Method 


RANGE  MAX=  395  366  MAX*  366  “IN*  355  “AX*  355  MIN*  365  MAX*  365  FIN*  290 


roportion  of  Advancements  Selected  by  Alternative  Weighting  Methods 
From  Each  Rank -Ordered  Quarter  for  Each  Component  (1?  Rate  Groups) 
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TABLE  12  (continued) 
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Pay  Grade 


Fig.  1.  Opinions  of  what  proportionate  influences  of  advancement  components 
actually  are  (Question  1). 


Fig.  3.  Opinions,  by  pay  grade,  of  fairness  of  alternative  sets  of  proportionate 
influences  of  advancement  components  (Questions  3-7) . 
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fairness  of  alte: 


Pay  Grade 


Fig.  S.  Opinions,  by  pay  grade,  of  fairness  of  an  alternative  set  of  propor¬ 
tionate  influences  of  advancement  components  (Question  5) 
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alternative  set  of  propor- 


Method  1 


Fig.  7.  Proportion  of  advancements  selected  from  each  rank-ordered  quarter  of  each  co^xment  as  related  to 
incidence.  (Rate:  ATI.  PNA  Incidence:  Present-Low;  Previous -Low.) 


Rate:  FTGC.  PNA  Incidence:  Present -High;  Previous- 


Coaponent  Quarter  -  Top  (T) .  2nd.  ird.  Bottom  (B) 

Fig.  8.  Proportion  of  advene earn is  selected  (ram  each  rank -ordered  quarter  of  each  coeaunnent  as  related  to 
PhA  incidence. 
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APPENDIX  B 


Abbreviations  and  Complete  Titles  of 
Navy  Ratings  Selected  for  Analysis 

ABH  -  Aviation  Boatswain's  Mate  H  (Aircraft  Handling) 

ADJ  -  Aviation  Machinist's  Mate  J  (Jet  Engine  Mechanic) 

ADR  -  Aviation  Machinist's  Mate  R  (Reciprocating  Engine  Mechanic) 

AK  -  Aviation  Storekeeper 

AT  -  Aviation  Electronics  Technician 

AX  -  Aviation  Antisubmarine  Warfare  (ASW)  Technician 

CS  -  Commissaryman 

DC  -  Damage  Controlman 

DT  -  Dental  Technician 

EM  -  Electrician's  Mate 

FT  -  Fire  Control  Technician 

HM  -  Hospital  Corpsman 

MM  -  Machinist's  Mate 

MN  -  Mineman 

PN  -  Personnelman 

RM  -  Radioman 

SD  -  Steward 

ST  -  Sonar  Technician 

TM  -  Torpedoman's  Mate 
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APPENDIX  C 

Questionnaire  on  Factors  in  Advancement 
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APPENDIX  C 


TyE  BASIS  OF  PROMOTION 

Proportionate  Influence  of  Factors  in 
the  Enlisted  Advancement  System 


INSTRUCTIONS:  At  the  direction  of  the  Chief  of  Naval  Personnel  this  Laboratory 
is  conducting  a  study  of  factors  in  the  Enlisted  Performance 
Evaluation  and  Advancement  Systems.  One  of  the  major  questions 
we  are  investigating  is  "What  should  be  the  basis  of  Advancement?" 
We  would  appreciate  your  judgment  on  this  also.  Please  indicate 
what  you  believe  the  relative  or  proportionate  influences  of 
various  factors  are  and  should  be,  for  advancement  U>  your  present 
rate  (exception:  SCPO's  and  MCPO's  indicate  your  opinion  as  it 
applies  to  advancement  from  E-6  t£  E-7  in  your  rating) . 
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T3 
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•h  rt 
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First,  for  Questions  1  and  2,  indicate  what  you  believe  the  relative  influence 
of  various  factors  actually  are  and  should  be  respectively  by  writing  in  the 
boxes  percentages  which 
total  100%.  For  example,  if 
you  believe  the  "Awards" 
factor  is  three  fourths  the 
influence,  and  "Time  in  Rate"! 
is  the  remaining  one  fourth 
influence,  you  would  write 
in  75%  and  25%  respectively 
for  those  two  factors  and 
00%  for  all  others. 
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1.  What  you  believe  the 
percentage  influences 
actually  are  for  your 
rate. 


Mill 


□  □  □  □  □  CDS 

(If  "other,"  indicate:  _ ) 


2.  What  you  think  the 
percentage  influences 
should  be. 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

100 

(If  "other,"  indicate:  _ ) 


Preceding  page  blank 


53 


APPENDIX  C  (continued) 


Next,  for  Questions  3-7,  indicate  how  fair  each  of  the  theoretical  combinations 
would  be  if  it  were  applied  as  the  basis  to r  advancement  to  your  rate: 

- Conpletely  Fair 


\\\\\%\% 

1  i  1  1  1  l  i  i 

3.  a  b  c  d  e  f  XA  j~T7~|  QT]  Q7]  Q7J  f~T6~|  (~I6~|  fool  f~100| 

4.  abcdef  XB  [~3Q~|  [~30~1  fl5~|  flS~|  f~5~l  |  S~1  fOQ~1  fTool 

5.  a  b  c  d  e  f  XC  [~43~|  [7i~\  fTTl  fTH  |  S~|  |~3~)  f~00~l  flOOl 

6.  abcdef  XD  |  90  |  |  4  |  |  j  1  |  2  |  |  1  |  |  T~1  I  66  |  |  1001 

7.  abcdef  xe  |~T5~1  RtTl  [T5~l  |~TF|  m  |  5~|  [~oo~|  fTool 


—  Mostly  Fair 

- Slightly  Fair 

I - Slightly  Unfair 

Ir- — Largely  Unfair 

p—Extremely  Unfair 


Theoretical 

Combination 
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Methodology  Employed  in  Computation  of 
Proportionate  Influences  of  Components 
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APPENDIX  D 


Methodology  Employed  in  Computation  of 
Proportionate  Influences  of  Components 


1.  Selection  of  an  appropriate  model  for  determining  the  proportionate 
influences  of  components  in  a  composite  has  been  the  subject  of  theo¬ 
rizing  and  research  for  many  years.  Among  others,  the  issue  is  nicely 
discussed  by  Ghiselli  (1964),  Darlington  (1968),  Dunnette  §  Hoggatt 
(1957),  and  Stanley  fi  Wang  (1968).  The  various  means  of  differentially 
weighting  components  include  the  following- -expressing  the  size  of  each 
component's : 

a.  Maximum/minimum  score. 

b.  Multiplier  weight  of  raw  score. 

c.  Standard  Deviation. 

d.  Correlation  coefficient-beta  weight  (£0)  products  of  regression 
analysis. 

The  Navy  enlisted  advancement  system  uses  a  combination  of  the 
first  and  second,  as  displayed  in  Table  1.  Thus  after  the  conponents 
are  "weighted"  by  various  multipliers,  they  are  added  to  form  a  single 
composite,  the  Final  Multiple  Score  (FMS) .  This  final  composite  is 
then  available  as  a  dependent  variable  to  perform  regression  analysis. 

2.  For  this  analysis,  the  use  of  the  above  first  and  second  methods  of 
representing  component  weights,  and  the  use  of  the  FMS  as  the  depend¬ 
ent  variable,  were  rejected  as  not  only  inappropriate,  but  quite  mis¬ 
leading,  for  the  following  reasons: 

a.  Relative  differences  in  standard  deviation  (the  third  of  the 
above  enumerated  methods)  are  a  much  greater  influence  among  components 
than  is  the  raw  multiplier  or  the  maximum/minimum  scores.  Intercorre¬ 
lations  among  the  components  are  another  source  of  influence. 

b.  Scores  in  the  high/low  tail  (maximum/minimum  score)  of  a  dis¬ 
tribution  are  quite  unstable  for  statistical  analysis.  There  could  be 
a  few  scores  there  or  a  substantial  pile-up  of  scores. 

c.  Use  of  the  FMS  as  the  dependent  variable  for  regression  anal¬ 
ysis  is  particularly  misleading  because  advancements  are  not  selected 
from  the  top  of  the  rank-ordered  FMS.  Instead,  advancements  are  se¬ 
lected  after  exam-fails  (a  substantial  majority  in  competition  for  Pay 
Grades  6  and  7)  are  deleted. 

3.  It  is  thus  important  to  distinguish  between  two  kinds  of  weights 
(Ghiselli,  1964).  The  nominal  weights  are  those  deliberately  assigned 
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to  the  componcnts--the  intended  policy  weights,  frequently  expressed 
simply  as  a  percentage  of  a  maximum  score.  The  effective  weights  are 
those  which  the  components  actually  carry  in  the  composite.  The  effec¬ 
tive  and  nominal  weights  may  or  may  not  be  the  same  (dependent  espe¬ 
cially  upon  the  standard  deviations  and  intercorrelations  among  the 
components  as  discussed  above),  and  usually  are  not  the  same. 

4.  The  only  relevant  criterion  for  a  question  of  proportionate  influ¬ 
ences  in  advancement  is  the  fact  of  advancement  itself.  Thus  the 
dichotomous  criterion  of  advanced-not  advanced  was  selected  as  the 
basis  of  this  study.  In  its  application  to  regression  analysis,  the 
following  considerations  and  adjustments  are  pertinent: 

a.  The  mathematical  model  for  regression  analysis  assumes  a  nor¬ 
mal  distribution  of  the  dependent  variable. 

b.  The  Navy  system  functions  as  a  "closed"  system,  i.e.,  all  com¬ 
ponents  are  accounted  for  so  that  the  percentage  influences  should 
total  100  percent.  In  regression  analysis  this  is  indicated  by: 

(1)  The  coefficient  of  multiple  correlation  (R)  equaling  1.0, 
and 

(2)  The  Sum  (I)  of  the  correlation  coefficient-beta  weight 
(rB)  products  equaling  R2,  in  turn  equaling  1.0. 

c.  Use  of  a  dichotomous  dependent  variable  will  substantially 
restrict  the  coefficient  R  below  1.0  which  requires  an  adjustment  of 
the  rB  products  by  an  increase  proportionate  to  the  increase  from  the 
restricted  R2  back  to  1.0.  The  method  is  illustrated  in  Table  3.  For 
example,  the  rB  product  for  the  Pay  Grade  4  exam  component  is  multi¬ 
plied  by  1.46,  the  amount  necessary  to  increase  the  R.2  from  .685  to 
1.0,  thereby  adjusting  the  rB  from  .596  to  .870;  and  all  adjusted  com¬ 
ponents  (.870,  .126,  .006  and  -.002)  totaling  1.000. 

d.  Due  to  rounding  to  nearest  whole  percentage,  and  distortion 
from  use  of  a  dichotomous  variable,  obtained  total  values  will  vary  a 
few  percentage  points  from  100  percent. 

e.  Due  to  the  dominance  of  one  component  to  the  total  exclusion 
of  another,  the  excluded  component  will  sometimes  assume  a  negative  rB 
value  from  the  effect  of  a  negative  correlation  with  the  dominant  com¬ 
ponent.  The  negative  rB  effect  is  to  be  distinguished  from  the  system 
causing  a  negative  weight  (i.e.,  points  are  not  systematically  sub¬ 
tracted  from  the  candidate  high  on  the  excluded  component). 
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